{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Beautifulsoup: find by regual expression\n",
    "**First we import re for regex. Then, open a [page](https://morvanzhou.github.io/static/scraping/table.html), then we can apply beautifulsoup on this page's html.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<!DOCTYPE html>\n<html lang=\"cn\">\n<head>\n\t<meta charset=\"UTF-8\">\n\t<title>爬虫练习 表格 table | 莫烦 Python</title>\n\n\t<style>\n\timg {\n\t\twidth: 250px;\n\t}\n\ttable{\n\t\twidth:50%;\n\t}\n\ttd{\n\t\tmargin:10px;\n\t\tpadding:15px;\n\t}\n\t</style>\n</head>\n<body>\n\n<h1>表格 爬虫练习</h1>\n\n<p>这是一个在 <a href=\"https://morvanzhou.github.io/\" >莫烦 Python</a> 的 <a href=\"https://morvanzhou.github.io/tutorials/scraping\" >爬虫教程</a>\n\t里无敌简单的网页, 所有的 code 让你一目了然, 清晰无比.</p>\n\n<br>\n<table id=\"course-list\">\n\t<tr>\n\t\t<th>\n\t\t\t分类\n\t\t</th><th>\n\t\t\t名字\n\t\t</th><th>\n\t\t\t时长\n\t\t</th><th>\n\t\t\t预览\n\t\t</th>\n\t</tr>\n\n\t<tr id=\"course1\" class=\"ml\">\n\t\t<td>\n\t\t\t机器学习\n\t\t</td><td>\n\t\t\t<a href=\"https://morvanzhou.github.io/tutorials/machine-learning/tensorflow/\">\n\t\t\t\tTensorflow 神经网络</a>\n\t\t</td><td>\n\t\t\t2:00\n\t\t</td><td>\n\t\t\t<img src=\"https://morvanzhou.github.io/static/img/course_cover/tf.jpg\">\n\t\t</td>\n\t</tr>\n\n\t<tr id=\"course2\" class=\"ml\">\n\t\t<td>\n\t\t\t机器学习\n\t\t</td><td>\n\t\t\t<a href=\"https://morvanzhou.github.io/tutorials/machine-learning/reinforcement-learning/\">\n\t\t\t\t强化学习</a>\n\t\t</td><td>\n\t\t\t5:00\n\t\t</td><td>\n\t\t\t<img src=\"https://morvanzhou.github.io/static/img/course_cover/rl.jpg\">\n\t\t</td>\n\t</tr>\n\n\t<tr id=\"course3\" class=\"data\">\n\t\t<td>\n\t\t\t数据处理\n\t\t</td><td>\n\t\t\t<a href=\"https://morvanzhou.github.io/tutorials/data-manipulation/scraping/\">\n\t\t\t\t爬虫</a>\n\t\t</td><td>\n\t\t\t3:00\n\t\t</td><td>\n\t\t\t<img src=\"https://morvanzhou.github.io/static/img/course_cover/scraping.jpg\">\n\t\t</td>\n\t</tr>\n\n</table>\n\n</body>\n</html>\n"
     ]
    }
   ],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "from urllib.request import urlopen\n",
    "import re\n",
    "\n",
    "# if has Chinese, apply decode()\n",
    "html = urlopen(\"https://morvanzhou.github.io/static/scraping/table.html\").read().decode('utf-8')\n",
    "print(html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Parsing this html using a method called lxml, create a soup object. Find all \"img\" tag which has a src in a given pattern.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://morvanzhou.github.io/static/img/course_cover/tf.jpg\nhttps://morvanzhou.github.io/static/img/course_cover/rl.jpg\nhttps://morvanzhou.github.io/static/img/course_cover/scraping.jpg\n"
     ]
    }
   ],
   "source": [
    "soup = BeautifulSoup(html, features='lxml')\n",
    "\n",
    "img_links = soup.find_all(\"img\", {\"src\": re.compile('.*?\\.jpg')})\n",
    "for link in img_links:\n",
    "    print(link['src'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Or using some helpful functions to find tags. Access the attribute of found tags using a key just like what you do in a python dictionary.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://morvanzhou.github.io/\nhttps://morvanzhou.github.io/tutorials/scraping\nhttps://morvanzhou.github.io/tutorials/machine-learning/tensorflow/\nhttps://morvanzhou.github.io/tutorials/machine-learning/reinforcement-learning/\nhttps://morvanzhou.github.io/tutorials/data-manipulation/scraping/\n"
     ]
    }
   ],
   "source": [
    "course_links = soup.find_all('a', {'href': re.compile('https://morvan.*')})\n",
    "for link in course_links:\n",
    "    print(link['href'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
